{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f891f81",
   "metadata": {},
   "source": [
    "## Dependency Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b7f8025",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "!pip install -q git+https://github.com/IBM/composite-adv.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acd260c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from composite_adv.utilities import make_dataloader\n",
    "\n",
    "# Load Dataset\n",
    "data_loader = make_dataloader('../data/', 'cifar10', batch_size=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15c212ed",
   "metadata": {},
   "source": [
    "`composite_adv.utilities.download_gdrive()` provides you an easy way to download checkpoint from Google Drive.\n",
    "```python\n",
    "!pip install -q --upgrade --no-cache-dir gdown\n",
    "from composite_adv.utilities import download_gdrive\n",
    "\n",
    "download_gdrive('1109eOxG5sSIxCwe_BUKRViMrSF20Ac4c', 'cifar10-resnet_50-gat_fs.pt')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8ecf34d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from composite_adv.utilities import make_model, robustness_evaluate\n",
    "\n",
    "# Load a model\n",
    "model = make_model('resnet50', 'cifar10', checkpoint_path='../cvpr23_camera_ready/cifar10/cifar10-resnet_50-gat_fs.pt')\n",
    "\n",
    "# Send to GPU\n",
    "import torch\n",
    "if not torch.cuda.is_available():\n",
    "    print('using CPU, this will be slow')\n",
    "else:\n",
    "    model.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71f03251",
   "metadata": {},
   "source": [
    "## CARBEN Leaderboard Evaluation\n",
    "\n",
    "\n",
    "We maintain the [leaderboards](https://hsiung.cc/CARBEN/) to track the progress of the **Compositional Adversarial Robustness**, and hereby solicit model submissions to compete against composite perturbations in our leaderboard. If you would like to submit your model, please evaluate your model using the following **four metrics** (Standard Accuracy, Auto Attack Robust Accuracy, Semantic Attacks Robust Accuracy and Full Attacks Robust Accuracy). After evalaution, please fill out the [Google Form](https://forms.gle/nNhPgQiq2jyyUCmA7), and we would periodically update the leaderboard after confirmation!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e768f32",
   "metadata": {},
   "source": [
    "### 1. Standard Accuracy (%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9fe4d42b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from composite_adv.attacks import NoAttack\n",
    "\n",
    "attack = NoAttack()\n",
    "robustness_evaluate(model, attack, data_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf74ced7",
   "metadata": {},
   "source": [
    "### 2. AutoAttack Robust Accuracy (%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cdc48bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from composite_adv.attacks import AutoLinfAttack\n",
    "attack = AutoLinfAttack(model, 'cifar10', bound=8/255)\n",
    "\n",
    "robustness_evaluate(model, attack, data_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf5ab58b",
   "metadata": {},
   "source": [
    "### 3. Semantic Attacks Robust Accuracy (%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd611f83",
   "metadata": {},
   "outputs": [],
   "source": [
    "from composite_adv.attacks import CompositeAttack\n",
    "attack = CompositeAttack(model, enabled_attack=(0,1,2,3,4), order_schedule=\"scheduled\")\n",
    "\n",
    "robustness_evaluate(model, attack, data_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55b3549d",
   "metadata": {},
   "source": [
    "### 4. Full Attacks Robust Accuracy (%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7a3b5b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from composite_adv.attacks import CompositeAttack\n",
    "attack = CompositeAttack(model, enabled_attack=(0,1,2,3,4,5), order_schedule=\"scheduled\")\n",
    "\n",
    "robustness_evaluate(model, attack, data_loader)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
